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II. RELATED APPEALS AND INTERFERENCES 

There are no known related appeals or interferences. 

III. STATUS OF CLAIMS 

1 . Claims 1 through 28 represent all claims currently pending in the 
application. 

2. Claims 1 through 28 are rejected. 

3. The rejection of claims 1 through 28 is hereby appealed. 

IV. STATUS OF AMENDMENTS 

An amendment to claims 1,12, and 21 was submitted by Appellant on July 5, 
2007 in response to the Final Office Action dated May 14, 2007. In accordance with the 
Advisory Action dated July 20, 2007, the amendment of July 5, 2007 was not entered 
by the Examiner. Consequently, for purposes of Appeal, claims 1-28 are in the form 
presented in the amendment filed by the Appellant on February 27, 2007 in response to 
the first Office Action issued in this case. These claims are presented in the attached 
Claims Appendix. No other amendments are currently pending. 

V. SUMMARY OF THE CLAIMED SUBJECT MATTER 

The pending patent application includes three independent claims: claims 1,12, 
and 21 . A summary of the subject matter claimed in each independent claim is 
provided below: 
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a. Subject Matter of Independent Claim 1 : 

In general, the subject matter of claim 1 relates to a technique for automatically 
constructing an image mosaic from a set of images of a scene by replacing occluded 
regions in an image with corresponding non-occluded regions from other images. 

As defined in claim 1 , the claimed computer executable instructions generally 
begin operation by "inputting a set of images of a scene," such as, for example, image 
frame 400 and 410, as illustrated by FIG. 4Aand FIG. 4B, respectively. See also 
element 200 of FIG. 2, which provides an image input module for receiving image 
frames of a scene either from one or more cameras (205) or from a database or 
directory of images (210). 

Once these images have been received, the claimed computer executable 
instructions continue by "registering the set of images" (see also the image registration 
module 215 of FIG. 2). As is generally well understood by those skilled in the art, image 
registration is a process wherein two or more images are rotated, scaled, warped, 
translated, etc., as necessary, so that the features between those images are aligned. 
This concept is described throughput the specification of the present application. For 
example, paragraph [0018] of the present application explains that the claimed 
computer executable instructions operate "... by first aligning or registering a set of 
two or more images of a scene. During alignment of the images, conventional image 
registration techniques, including, for example, translation, rotation, scaling, and 
perspective warping of the images are used to align the images..." 

Next, the claimed computer executable instructions operate to compare "the set 
of images to identify areas of difference between the images for all images, said areas 
of difference representing regions of potential occlusion in each image." In other 
words, given the set of registered images, those images are examined to identify areas 
of each image that differ from one image to the next. Any areas in one image that are 
different than corresponding areas in any other images are identified as corresponding 



Appeal Brief for U.S. Application 10/606,002 
Page 4 of 37 

to "regions of potential occlusion." It should be noted here that by identifying any region 
in a particular image as being a region of potential occlusion, all corresponding regions 
in all other images in the set of registered images are also identified as corresponding 
regions of potential occlusion. See for example, element 260 of FIG. 2, which provides 
an "image comparison module" that compares aligned images to identify areas of 
difference between those images. 

Note that the concept of "occlusion" in images is generally well understood by 
those skilled in the art as referring to regions of an image that are obscured from view. 
For example, as described in paragraph [0056] of the present application (US Patent 
Application No. 2004-0264806 A1 ), "one might have several images of a famous 
monument, but be unable to capture an image of the scene which does not have one or 
more areas of occlusion due to people wandering in and out of the image frame." This 
example is further illustrated by FIG. 4A and FIG. 4B, which illustrate a pictorial 
representation of a sequence of two image frames having areas of occlusions. For 
example, as described in paragraph [0107] of the present application, as "illustrated by 
FIG. 4A, image frame 400 illustrates an arch 420 that is partially occluded by two people 
430. Similarly, as illustrated by FIG. 4B, image frame 410 illustrates the arch 420, with a 
background of the image frame being partially occluded by a third person 440." 

At this point in the claimed technique, there is a set of aligned or registered 
images having corresponding regions of potential occlusion. However, while a region of 
potential occlusion in one image may represent an actual region of occlusion relative to 
the scene, that same region in another of the images may correspond to a non- 
occluded region of the same scene. Consequently, the next step performed by the 
claimed computer executable instructions is to examine each of the potential regions of 
occlusion in each image to determine whether those potentially occluded regions are 
actually occluded or not. 

In particular, the claimed computer executable instructions operate to determine, 
"for each image, whether regions of potential occlusion in each image represent actual 
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regions of occlusion, or whether the regions of potential occlusion in each image 
represent regions of non-occlusion, by determining a level of discontinuity along 
an exterior border of each region of potential occlusion in each image." In other 
words, having identified potential regions of occlusion for all images, the exterior 
borders of those regions are then examined to determine a level of discontinuity along 
those borders. The level of discontinuity determined for each particular border is then 
used in determining whether each of the corresponding regions is an actual region of 
occlusion or not. See for example, element 270 of FIG. 2, which provides a 
"discontinuity detection module" that compares "levels of discontinuity in two or more 
images for a particular potentially occluded region. Then, the image having a larger 
amount of discontinuity along the border of the potentially occluded region is identified 
as the image containing the occlusion" (see paragraph [0067] of the present 
application). 

Finally, having made a determination as to which regions in each image are 
areas of actual occlusion and which areas of each image are non-occluded regions, 
the claimed computer executable instructions operates to create a mosaic image "by 
replacing at least one actual region of occlusion in one image from the set of images 
with corresponding regions of non-occlusion from at least one other image from the set 
of images of the scene." 

A simple graphical illustration of the technique defined by claim 1 is illustrated by 
the sequence of images shown in FIG. 4A through FIG. 9, as discussed in paragraphs 
[01 06] through [01 1 0] of the present application. In particular, these figures, in 
combination with paragraphs [0106] through [0110], which show a non-occluded mosaic 
image (FIG. 9) being constructed from two input images (400 and 410 of FIG 4A and 
FIG. 4B, respectively), with a region of actual occlusion (440 of FIG. 4B) in one of the 
images (410 of FIG. 4B) being replaced by a corresponding region taken from the other 
image (400 of FIG. 4A). 
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b. Subject Matter of Independent Claim 12: 

In general, the subject matter of claim 12 relates to a system for removing 
occlusions from a composite image formed from a set of images of a scene. 

As defined by claim 12, the claimed system begins operation by acquiring a set 
of images of a scene. These images are then aligned relative to a base image selected 
from the set of images. As is generally well understood by those skilled in the art, 
image alignment is a process wherein two or more images are rotated, scaled, warped, 
translated, etc., as necessary, so that the features between those images are aligned or 
registered. This concept is described throughput the specification of the present 
application. For example, paragraph [001 8] of the present application explains that the 
claimed computer executable instructions operate "... by first aligning or registering a 
set of two or more images of a scene. During alignment of the images, conventional 
image registration techniques, including, for example, translation, rotation, scaling, and 
perspective warping of the images are used to align the images..." 

Next, the aligned images are examined to identify areas of potential occlusion in 
each of the aligned images. Note that the concept of "occlusion" in images is generally 
well understood by those skilled in the art as referring to regions of an image that are 
obscured from view. For example, as described in paragraph [0056] of the present 
application (US Patent Application No. 2004-0264806 A1 ), "one might have several 
images of a famous monument, but be unable to capture an image of the scene which 
does not have one or more areas of occlusion due to people wandering in and out of the 
image frame." This example is further illustrated by FIG. 4A and FIG. 4B, which 
illustrate a pictorial representation of a sequence of two image frames having areas of 
occlusions. For example, as described in paragraph [0107] of the present application, 
as "illustrated by FIG. 4A, image frame 400 illustrates an arch 420 that is partially 
occluded by two people 430. Similarly, as illustrated by FIG. 4B, image frame 410 
illustrates the arch 420, with a background of the image frame being partially occluded 
by a third person 440." 
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Once the areas of potential occlusion in each image have been identified, the 
claimed system continues operation by selecting a one of the images from the set of 
images. This selected images is referred to as a "seed image" since it is later used a 
foundation from which the composite image is formed. In particular, given the selected 
seed image, the claimed system examines that seed image to determine "whether each 
area of potential occlusion in the seed image is an actual area of occlusion by 
examining each area of potential occlusion in the seed image to determine whether a 
level of discontinuity along an outer edge of each area of discontinuity exceeds a 
predetermined threshold." 

In other words, the outer edges of each area of potential occlusion in the seed 
image are examined to determine a level of discontinuity along those edges. If the level 
of discontinuity for any edge exceeds some predetermined threshold, than the 
corresponding area of potential occlusion is identified as an actual area of occlusion. 
See for example, element 270 of FIG. 2, which provides a "discontinuity detection 
module" that examines "...the pixels close to the border of each potential area of 
occlusion to identify pixel color or intensity discontinuities perpendicular to those 
edges..." (see paragraph [0067] of the present application). Again, when the resulting 
level of discontinuity exceeds the predetermined threshold, the region is identified as an 
actual area of occlusion. 

Finally, the claimed system operates by "replacing areas of actual occlusion in 
the seed image with corresponding non-occluded areas from one of the other images in 
the set to form a composite image from the seed image." 

A simple graphical illustration of the system defined by claim 12 is illustrated by 
the sequence of images shown in FIG. 4A through FIG. 9, as discussed in paragraphs 
[01 06] through [01 1 0] of the present application. In particular, these figures, in 
combination with paragraphs [0106] through [01 10], which show a non-occluded 
composite image (FIG. 9) being constructed from two input images (400 and 410 of FIG 
4A and FIG. 4B, respectively), with an area of actual occlusion (440 of FIG. 4B) in a 



Appeal Brief for U.S. Application 10/606,002 
Page 8 of 37 

"seed image" (410 of FIG. 4B) being replaced by a corresponding region taken from the 
other image (400 of FIG. 4A). 

c. Subject Matter of Independent Claim 21 : 

In general, the subject matter of claim 21 relates to process for removing 
occlusions from a mosaic image created from a set of images of a scene. 

As defined by claim 21 , the claimed process begins operation by inputting a set 
of two or more images of a scene. These images are then aligned relative to a base 
image selected from the set of images. As is generally well understood by those skilled 
in the art, image alignment is a process wherein two or more images are rotated, 
scaled, warped, translated, etc., as necessary, so that the features between those 
images are aligned or registered. This concept is described throughput the specification 
of the present application. For example, paragraph [001 8] of the present application 
explains that the claimed computer executable instructions operate "... by first aligning 
or registering a set of two or more images of a scene. During alignment of the images, 
conventional image registration techniques, including, for example, translation, rotation, 
scaling, and perspective warping of the images are used to align the images..." 

Next, the aligned images are compared "to identify areas of potential occlusion in 
each of the aligned images." See for example, element 260 of FIG. 2, which provides 
an "image comparison module" that compares aligned images to identify areas of 
difference between those images. Note that the concept of "occlusion" in images is 
generally well understood by those skilled in the art as referring to regions of an image 
that are obscured from view. For example, as described in paragraph [0056] of the 
present application (US Patent Application No. 2004-0264806 A1), "one might have 
several images of a famous monument, but be unable to capture an image of the scene 
which does not have one or more areas of occlusion due to people wandering in and 
out of the image frame." This example is further illustrated by FIG. 4A and FIG. 4B, 
which illustrate a pictorial representation of a sequence of two image frames having 
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areas of occlusions. For example, as described in paragraph [0107] of the present 
application, as "illustrated by FIG. 4A, image frame 400 illustrates an arch 420 that is 
partially occluded by two people 430. Similarly, as illustrated by FIG. 4B, image frame 
410 illustrates the arch 420, with a background of the image frame being partially 
occluded by a third person 440." 

Once the areas of potential occlusion in each image have been identified, the 
claimed process continues operation by "determining a level of discontinuity along 
an outer edge of each area of potential occlusion for each image, said level of 
discontinuity indicating an area of actual occlusion where the level of discontinuity 
exceeds a predetermined discontinuity threshold, and said level of discontinuity 
indicating an area of non-occlusion where the level of discontinuity is less than the 
predetermined discontinuity threshold." 

In other words, the outer edges of each area of potential occlusion of each image 
are examined to determine a level of discontinuity along those edges. If the level of 
discontinuity for any edge exceeds predetermined discontinuity threshold, than the 
corresponding area of potential occlusion is identified as an area of actual occlusion. 
Conversely, if the level of discontinuity is less than the predetermined discontinuity 
threshold, than the corresponding area of potential occlusion is identified as an area of 
non-occlusion. See for example, element 270 of FIG. 2, which provides a 
"discontinuity detection module" that examines "...the pixels close to the border of each 
potential area of occlusion to identify pixel color or intensity discontinuities perpendicular 
to those edges..." (see paragraph [0067] of the present application). 

Next, once each of the areas of potential occlusion in each image are identified 
as either an area of actual occlusion or as an area of non-occlusion, the claimed 
process constructs an image mask for each image that specifically identifies areas of 
actual occlusion and areas of non-occlusion in each image. See for example, elements 
260 and 270 of FIG. 2, wherein the image comparison module (260) creates an initial 
image mask (265) that is subsequently updated by the discontinuity detection module 
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(270) when the areas of potential occlusion in each image are identified as either an 
area of actual occlusion or as an area of non-occlusion (see paragraph [0066] and 
[0067] of the present application). 

Finally, the claimed process uses the image masks constructed for each image 
to create "a mosaic image by replacing areas of actual occlusion in one of the images 
with corresponding areas of non-occlusion from one of the other images. 

A simple graphical illustration of the process defined by claim 21 is illustrated by 
the sequence of images shown in FIG. 4A through FIG. 9, as discussed in paragraphs 
[01 06] through [01 1 0] of the present application. In particular, these figures, in 
combination with paragraphs [0106] through [01 10], which show a non-occluded mosaic 
image (FIG. 9) being constructed from two input images (400 and 410 of FIG 4A and 
FIG. 4B, respectively), with an area of actual occlusion (440 of FIG. 4B) in one image 
(410 of FIG. 4B) being replaced by a corresponding region taken from the other image 
(400 of FIG. 4A) as a function of an updated image mask (element 700 of FIG. 7) for 
image frame 410 of FIG. 4B. 

VI. GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

a. Independent claim 12 stands rejected under 35 U.S. C. §1 02(e) as being 
anticipated by U.S. Patent Application Publication No. 20020181762 to Silber 
(hereinafter "Silber"). 

b. Independent claim 1 stands rejected under 35 U.S.C. §1 03(a) as being 
unpatentable over Silber in view of U.S. Patent No. 6,366,316 to Parulski, etal. 
(hereinafter "Parulski). 

c. Independent claim 21 stands rejected under 35 U.S.C. §1 03(a) as being 
unpatentable over Silber in view of Parulski. 
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VII. ARGUMENT 

a. Rejection of Claim 12 under 35 U.S.C. §102(e): 

Independent claim 12 was rejected under 35 U.S.C. §1 02(e), as being 
anticipated by U.S. Patent Application Publication No. 20020181762 to Silber 
(hereinafter "Silber"). In view of the following discussion, the Appellant will show that 
one or more elements of the Appellant's claimed system are not disclosed by the cited 
art, and that the Appellants claimed system is therefore patentable over that cited art. 

In general, the Office Action rejected independent claim 12 under 35 USC 
§1 02(e) based on the rationale that the Silber reference teaches the Appellants' 
claimed "... system for removing occlusions from a composite image formed from a set 
of images of a scene..." 

In particular, with respect to the issue of "occlusions," the Office Action suggests 
that the Silber reference discloses "identifying areas of potential occlusions in each of 
the aligned images..." However, Appellant believes that the Office Action has 
misinterpreted the capabilities and features of the Silber reference with respect to these 
alleged occlusion identification capabilities. 

Specifically, as is well known to those skilled in the art, an occlusion in an image 
occurs when an object in an image frame of a particular scene covers or occludes all or 
part of another object in that image frame. In other words, an occlusion in an image 
exists when a region or area of a scene is hidden from view or otherwise covered from 
view by means of some intervening entity or object relative to a viewpoint from which an 
image of the scene is captured. More simply, the well understood concept of an 
occlusion in an image can be characterized as a region in an image that is obscured 
from view. The specification of the present application specifically defines this idea and 
is wholly consistent with the conventional interpretation of the term "occlusion" as 
recited above. 
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For example, as described in paragraph [0056] of the present application (US 
Patent Application No. 2004-0264806 A1), "one might have several images of a famous 
monument, but be unable to capture an image of the scene which does not have one or 
more areas of occlusion due to people wandering in and out of the image frame." This 
example is further illustrated by FIG. 4A and FIG. 4B, which illustrate a pictorial 
representation of a sequence of two image frames having areas of occlusions. 
Specifically, as described in paragraph [0107] of the present application, as "illustrated 
by FIG. 4A, image frame 400 illustrates an arch 420 that is partially occluded by two 
people 430. Similarly, as illustrated by FIG. 4B, image frame 410 illustrates the arch 
420, with a background of the image frame being partially occluded by a third person 
440." 

In contrast to the clear definitions and drawings provided in the present 
application, the Examiner has suggested that the interpretation of the term "occlusion" 
offered by the Appellant is only "one interpretation of the word occlusion..." and that if 
"the applicant intends to further define the region of occlusion to mean an area in which 
an object covers another object, the applicant is invited to do so." In response, the 
Appellant did in fact attempt to amend the claim to recite: "wherein an occlusion 
represents an area in any of the images that is at least partially obstructed from 
view by one or more objects..." However, in the Advisory Action, the Examiner 
responded by refusing to enter the above-quoted amendment and suggesting that the 
"applicant has amended the claims to encompass a different scope and thus would 
require further consideration and search." 

In view of both the Final Office Action and in view of the Advisory Action, it 
should be clear that the Examiner is interpreting the term "occlusion" in a manner that is 
in direct conflict with the commonly understood meaning of occlusions as specifically 
defined and graphically illustrated in the present specification. Specifically, the 
Examiner has insisted on redefining the term "occlusion" in a manner that is wholly 
inconsistent with both the present specification and drawings, and with the common 
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usage of occlusions in the art of image processing as it is generally understood to those 
skilled in the art. 

In fact, in paragraph 7 of the Final Office Action dated May 14, 2007, the 
Examiner states: "The examiner agrees that Silber does not completely provide a 
technique for deciding whether a particular edge represents the boundary of an 
object that is occluding some portion of the scene. .." As such, the meaning of the 
term "occlusion" as specifically defined by the Appellant in the specification of the 
present application and as clearly illustrated by the Appellant in the drawings of the 
present application is of primary concern in this Appeal. 

Specifically, in the Final Office Action, the Examiner argues: 

"... One of the definitions of "occlusion" is "the process of occluding," 
according to dictionary.com, where "occlude" means "to obstruct", or "to 
prevent the passage of." Any region that is masked or not used in further 
processing is an occluded region since the region is obstructed from 
further processing. The region that is not masked is used in further 
processing can also be interpreted as occluded, because it is occluded by 
the mask..." 

The Examiner then uses the circular logic of the above quoted argument in an 
attempt to explain that the Silber reference completely discloses the claimed elements 
of "identifying areas of potential occlusion in each of the aligned images. .." and 
"determining whether each area of potential occlusion in the seed image is an 
actual area of occlusion by examining each area of potential occlusion in the seed 
image to determine whether a level of discontinuity along an outer edge of each 
area of discontinuity exceeds a predetermined threshold. .." 

In particular, the Examiner argues that in FIG. 5, step 730, and in paragraph 
[0089], Silber teaches "determining whether each area of potential occlusion in the 
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seed image is an actual area of occlusion by examining each area of potential occlusion 
in the seed image to determine whether a level of discontinuity along an outer edge of 
each area of discontinuity exceeds a predetermined threshold." Specifically, in the 
Office Action, dated December 13, 2006, the Examiner stated that this feature of the 
Appellant's claimed system is taught by Silber "since the amount of focus of the line, 
which is equivalent to discontinuous because it is not a solid, dear line, is 
compared to a threshold. . ." (emphasis added). The Examiner then further clarified 
this position in the Final Office Action dated May 14, 2007 by arguing the following: 

"By determining whether a particular edge represents a best level 
focus for that edge, the edge is "highlighted" so to speak as a good area 
to represent the composite image. Therefore, a determination of a 
region of occlusion is accomplished by the evaluation of detected 
edges since the detected edges are evaluated, and a region is 
determined as occluded since certain areas are not further 
processed, the areas that are not of the valid edge, and thus occluded 
from further processing." (emphasis added) 

However, in contrast to the position argued by the Examiner, Appellant 
respectfully suggests that Silber does not disclose the elements suggested by the 
Office Action, regardless of how the term "occlusion" is interpreted . For example, 
as described by Silber in the Abstract, in independent claims 1 , 20, 21 and 22, and 
throughout the specification with respect to Figures 1-13, Silber is directed towards 
compositing images of single objects or "workpieces" in the case where each of the 
multiple images of the single objects is captured at a different depth of focus. In fact, 
Silber makes use of different depths of focus, in every disclosed and claimed 
embodiment, in order to capture 3D details of the single workpiece such that each 
feature of the single workpiece is captured at various levels of focus. The edges of 
each image are then examined to select those edges that are in best focus, with 
the best focused edges then being used to construct a composite image of the 
single workpiece. Clearly, Silber discloses a process for evaluating edge focus levels 
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to determine whether an edge is adequately focused in view of some predetermined 
focus threshold (e.g., the "numerical threshold value" described by Silber). 

For example, in paragraph [0016], Silber specifically summarizes his invention 
as follows: 

"[0016] This invention separately provides systems and methods that 
construct a desirable composite image while suppressing likely 
artifacts arising from out-of-focus features ." 

Further, in paragraph [0089], as cited by the Office Action with respect to Figure 
3, Silber teaches the following: 

"[0089] In step S730, a determination is made whether the selected 
source image analysis result is indicative of a valid edge. For example, if 
the source image analysis result is a numerical edge-focus indicator 
result, this result is compared to a numerical threshold value, where 
only values exceeding the threshold value are taken to be indicative 
of a valid and adequately-focused edge for purposes of constructing 
the composite image. If the source image analysis result is not 
indicative of a valid edge, then operation returns to step S715. Otherwise, 
operation continues to step S735." (emphasis added) 

Clearly, the focus-based edge detection process disclosed by Silber fails 
completely to provide a technique for first "identifying areas of potential occlusion..." 
and then determining whether each of those areas of potential occlusion is "an actual 
area of occlusion..." based on "whether a level of discontinuity along an outer 
edge of each area of discontinuity exceeds a predetermined threshold. . ." 

In fact, the invention described by Silber is specifically directed towards 
constructing a composite image from multiple images of a single object or workpiece 
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based solely on which edges are in best focus. Consequently, Appellant respectfully 
suggests that any composite image created by the Silber invention from a set of images 
that include one or more actual areas of occlusion would tend to include portions of 
both occluding objects and non-occluded regions of images as a function of edge 
focus levels rather than whether or not an area was actually occluded. In fact, 
Appellant believes that the edges of occluding objects would be favored by the Silber 
system since the edges of any occluding object in a particular image frame would 
necessarily be in better focus than any edges that are actually occluded and would 
therefore not even be visible. 

Clearly, Silber fails completely to provide any technique whatsoever for making a 
determination as to whether an area of an image is occluded, and then to selectively 
replace any occluded areas with corresponding non-occluded regions from other 
images. Therefore, in view of the preceding discussion, it is clear that the present 
system, as claimed by independent claim 12 has elements not disclosed in the Silber 
reference. Consequently, the rejection of claim 12 under 35 U.S.C. §1 02(e) is not 
proper. Therefore, the Appellants respectfully traverse the rejection of claim 12 under 
35 U.S.C. §1 02(e) in view of the novel language of claim 12. In particular, claim 12 
recites the following novel language: 

"A system for removing occlusions from a composite image formed 
from a set of images of a scene, comprising: 

acquiring at least two images of a scene from approximately the 
same viewpoint; 

aligning each of the images to a base image selected from the set 
of images; 

identifying areas of potential occlusion in each of the aligned 
images, 

selecting a seed image from the set of images; 
determining whether each area of potential occlusion in the 
seed image is an actual area of occlusion by examining each area of 
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potential occlusion in the seed image to determine whether a level of 
discontinuity along an outer edge of each area of discontinuity 
exceeds a predetermined threshold, 

replacing areas of actual occlusion in the seed image with 
corresponding non-occluded areas from one of the other images in the 
set to form a composite image from the seed image." (emphasis added) 

b. Rejection of Claim 1 under 35 U.S.C. §103(a): 

Independent claim 1 was rejected under 35 U.S.C. §1 03(a) as being 
unpatentable over Silber \n view of U.S. Patent No. 6,366,316 to Parulski, etal. 
(hereinafter "Parulski'). 

In general, in the Final Office Action dated May 14, 2007, the Examiner rejected 
independent claim 1 under 35 USC §1 03(a) based on the rationale that the Silber 
reference discloses the claimed computer executable instructions with the exception of 
"comparing the set of images to identify areas of difference between the images for all 
images, said areas of difference representing regions of potential occlusion in each 
image." The Office Action then continues by suggesting that this feature lacking in the 
Silber reference is disclosed by the Parulski reference. Further, Examiner specifically 
explains the functionality of the proposed Silber-Parulski combination reference as 
follows: 

"First, the proposed Silber-Parulski combination will receive two images, 
and then the second image will be subtracted from the first image to 
provide a single image of a 'subject,' representing a region of occlusion. 
This first step represents the Parulski contribution to the proposed 
Silber-Parulski combination. Next, the proposed Silber-Parulski 
combination will determine whether the regions of potential occlusion 
represent actual regions of occlusion by examining edges, as the 
prior art and prior rejection have stated, as having a valid focus . If it 
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does then a mosaic image is made with the two images by combining 
images." (emphasis added) 

Clearly, the Examiner has explained that the Parulski contribution to the 
proposed combination reference is limited to receiving "two images, and then the 
second image will be subtracted from the first image to provide a single image of a 
'subject,' representing a region of occlusion..." As such, one important issue is whether 
the Silber reference discloses the claimed elements pertaining to identification of 
regions of potential occlusion followed by a determination of whether each region 
potential occlusion is an actual region of occlusion or is a region of non-occlusion by 
determining a level of discontinuity along an exterior border of each region of potential 
occlusion. 

Thus, with respect to the issue of "occlusions," the Office Action suggests that 
the Silber reference discloses "determining, for each image, whether regions of 
potential occlusion in each image represent actual regions of occlusion, or 
whether the regions of potential occlusion in each image represent regions of non- 
occlusion, by determining a level of discontinuity along an exterior border of each 
region of potential occlusion in each image (fig. 5, step 730, as explained above)..." 
(emphasis added). However, Appellant believes that the Office Action has 
misinterpreted the capabilities and features of the Silber reference with respect to these 
alleged occlusion identification capabilities. 

Specifically, as is well known to those skilled in the art, an occlusion in an image 
occurs when an object in an image frame of a particular scene covers or occludes all or 
part of another object in that image frame. In other words, an occlusion in an image 
exists when a region or area of a scene is hidden from view or otherwise covered from 
view by means of some intervening entity or object relative to a viewpoint from which an 
image of the scene is captured. More simply, the well understood concept of an 
occlusion in an image can be characterized as a region in an image that is obscured 
from view. The specification of the present application specifically defines this idea and 
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is wholly consistent with the conventional interpretation of the term "occlusion" as 
recited above. 

For example, as described in paragraph [0056] of the present application (US 
Patent Application No. 2004-0264806 A1), "one might have several images of a famous 
monument, but be unable to capture an image of the scene which does not have one or 
more areas of occlusion due to people wandering in and out of the image frame." This 
example is further illustrated by FIG. 4A and FIG. 4B, which illustrate a pictorial 
representation of a sequence of two image frames having areas of occlusions. 
Specifically, as described in paragraph [0107] of the present application, as "illustrated 
by FIG. 4A, image frame 400 illustrates an arch 420 that is partially occluded by two 
people 430. Similarly, as illustrated by FIG. 4B, image frame 410 illustrates the arch 
420, with a background of the image frame being partially occluded by a third person 
440." 

In contrast to the clear definitions and drawings provided in the present 
application in the Final Office Action, the Examiner suggested that the interpretation of 
the term "occlusion" offered by the Appellant is only "one interpretation of the word 
occlusion..." and that if "the applicant intends to further define the region of occlusion to 
mean an area in which an object covers another object, the applicant is invited to do 
so." In response, the Appellant did in fact attempt to amend the claim to recite: 
"wherein an occlusion represents an area in any of the images that is at least 
partially obstructed from view by one or more objects. .." However, in the Advisory 
Action, the Examiner responded by refusing to enter the above-quoted amendment and 
suggesting that the "applicant has amended the claims to encompass a different scope 
and thus would require further consideration and search." 

In view of both the Final Office Action and in view of the Advisory Action, it 
should be clear that the Examiner is interpreting the term "occlusion" in a manner that is 
in direct conflict with the commonly understood meaning of occlusions as specifically 
defined and graphically illustrated in the present specification. Specifically, the 
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Examiner has insisted on redefining the term "occlusion" in a manner that is wholly 
inconsistent with both the present specification and drawings, and with the common 
usage of occlusions in the art of image processing as it is generally understood to those 
skilled in the art. 

In fact, in paragraph 7 of the Final Office Action dated May 14, 2007, the 
Examiner states: "The examiner agrees that Silber does not completely provide a 
technique for deciding whether a particular edge represents the boundary of an 
object that is occluding some portion of the scene. .." As such, the meaning of the 
term "occlusion" as specifically defined by the Appellant in the specification of the 
present application and as clearly illustrated by the Appellant in the drawings of the 
present application is of primary concern in this Appeal. 

Specifically, in the Final Office Action, the Examiner argues: 

"... One of the definitions of "occlusion" is "the process of occluding," 
according to dictionary.com, where "occlude" means "to obstruct", or "to 
prevent the passage of." Any region that is masked or not used in further 
processing is an occluded region since the region is obstructed from 
further processing. The region that is not masked is used in further 
processing can also be interpreted as occluded, because it is occluded by 
the mask..." 

The Examiner then uses the circular logic of the above quoted argument in an 
attempt to explain that the Silber reference completely discloses the claimed element of 
"determining, for each image, whether regions of potential occlusion in each image 
represent actual regions of occlusion, or whether the regions of potential occlusion 
in each image represent regions of non-occlusion, by determining a level of 
discontinuity along an exterior border of each region of potential occlusion in 
each image..." 
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In particular, the Examiner argues that in FIG. 5, step 730, and in paragraph 
[0089], Silber teaches "determining whether each area of potential occlusion in the 
seed image is an actual area of occlusion by examining each area of potential occlusion 
in the seed image to determine whether a level of discontinuity along an outer edge of 
each area of discontinuity exceeds a predetermined threshold." Specifically, in the 
Office Action, dated December 13, 2006, the Examiner stated that this feature of the 
Appellant's claimed system is taught by Silber "since the amount of focus of the line, 
which is equivalent to discontinuous because it is not a solid, clear line, is 
compared to a threshold..." (emphasis added). The Examiner then further clarified 
this position in the Final Office Action dated May 14, 2007 by arguing the following: 

"By determining whether a particular edge represents a best level 
focus for that edge, the edge is "highlighted" so to speak as a good area 
to represent the composite image. Therefore, a determination of a 
region of occlusion is accomplished by the evaluation of detected 
edges since the detected edges are evaluated, and a region is 
determined as occluded since certain areas are not further 
processed, the areas that are not of the valid edge, and thus occluded 
from further processing." (emphasis added) 

However, in contrast to the position argued by the Examiner, Appellant 
respectfully suggests that Silber does not disclose the elements suggested by the 
Office Action, regardless of how the term "occlusion" is interpreted . For example, 
as described by Silber in the Abstract, in independent claims 1 , 20, 21 and 22, and 
throughout the specification with respect to Figures 1-13, Silber is directed towards 
compositing images of single objects or "workpieces" in the case where each of the 
multiple images of the single objects is captured at a different depth of focus. In fact, 
Silber makes use of different depths of focus, in every disclosed and claimed 
embodiment, in order to capture 3D details of the single workpiece such that each 
feature of the single workpiece is captured at various levels of focus. The edges of 
each image are then examined to select those edges that are in best focus, with 
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the best focused edges then being used to construct a composite image of the 
single workpiece. Clearly, Silber discloses a process for evaluating edge focus levels 
to determine whether an edge is adequately focused in view of some predetermined 
focus threshold (e.g., the "numerical threshold value" described by Silber). 

For example, in paragraph [0016], Silber specifically summarizes his invention 
as follows: 

"[0016] This invention separately provides systems and methods that 
construct a desirable composite image while suppressing likely 
artifacts arising from out-of-focus features ." 

Further, in paragraph [0089], as cited by the Office Action with respect to Figure 
3, Silber teaches the following: 

"[0089] In step S730, a determination is made whether the selected 
source image analysis result is indicative of a valid edge. For example, if 
the source image analysis result is a numerical edge-focus indicator 
result, this result is compared to a numerical threshold value, where 
only values exceeding the threshold value are taken to be indicative 
of a valid and adeguately-focused edge for purposes of constructing 
the composite image. If the source image analysis result is not 
indicative of a valid edge, then operation returns to step S715. Otherwise, 
operation continues to step S735." (emphasis added) 

Clearly, the focus-based edge detection process disclosed by Silber fails 
completely to provide a technique for "determining... whether regions of potential 
occlusion... represent actual regions of occlusion... or ... regions of non-occlusion, by 
determining a level of discontinuity along an exterior border of each region of potential 
occlusion..." as disclosed and claimed by the Appellant. 
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In fact, the invention described by Silber \s specifically directed towards 
constructing a composite image from multiple images of a single object or workpiece 
based solely on which edges are in best focus. Consequently, Appellant respectfully 
suggests that any composite image created by the Silber invention from a set of images 
that include one or more actual areas of occlusion would tend to include portions of 
both occluding objects and non-occluded regions of images as a function of edge 
focus levels rather than whether or not an area was actually occluded. In fact, 
Appellant believes that the edges of occluding objects would be favored by the Silber 
system since the edges of any occluding object in a particular image frame would 
necessarily be in better focus than any edges that are actually occluded and would 
therefore not even be visible. 

Clearly, the proposed Silber- Parulski combination reference fails completely to 
provide any technique whatsoever for comparing images to identify areas of difference 
representing regions of potential occlusion in each image and then determining a level 
of discontinuity along an exterior border of each region of potential occlusion to 
determine whether each region represents an actual region of occlusion for use in 
creating a mosaic image by replacing actual region of occlusions with corresponding 
regions of non-occlusion from other images as disclosed and claimed by the Appellant. 
Therefore, in view of the preceding discussion, it is clear that the present system, as 
claimed by independent claim 1 has elements not disclosed in the proposed Silber- 
Parulski combination reference. Consequently, the rejection of claim 1 under 35 
U.S.C. §1 03(a) is not proper. Therefore, the Appellants respectfully traverse the 
rejection of claim 1 under 35 U.S.C. §1 03(a) in view of the novel language of claim 1. In 
particular, claim 1 recites the following novel language: 

"A physical computer-readable medium having stored thereon 
computer executable instructions for automatically constructing an image 
mosaic from a set of images of a scene, said computer executable 
instructions comprising: 

inputting a set of images of a scene; 
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registering the set of images; 

comparing the set of images to identify areas of difference 
between the images for all images, said areas of difference representing 
regions of potential occlusion in each image, 

determining, for each image, whether regions of potential 
occlusion in each image represent actual regions of occlusion, or 
whether the regions of potential occlusion in each image represent regions 
of non-occlusion, by determining a level of discontinuity along an 
exterior border of each region of potential occlusion in each image; 
and 

creating a mosaic image by replacing at least one actual region 
of occlusion in one image from the set of images with corresponding 
regions of non-occlusion from at least one other image from the set of 
images of the scene." (emphasis added) 

c. Rejection of Claim 21 under 35 U.S.C. 5103(a): 

Independent claim 21 was rejected under 35 U.S.C. §1 03(a) as being 
unpatentable over Silber \n view of U.S. Patent No. 6,366,316 to Parulski, etal. 
(hereinafter "Parulski ). 

In general, in the Final Office Action dated May 14, 2007, the Examiner rejected 
independent claim 21 under 35 USC §1 03(a) based on the rationale that the Silber 
reference discloses the claimed process when combined with the Parulski reference. 
Further, Examiner specifically explains the functionality of the proposed Silber-Parulski 
combination reference as follows: 

"First, the proposed Silber-Parulski combination will receive two images, 
and then the second image will be subtracted from the first image to 
provide a single image of a 'subject,' representing a region of occlusion. 
This first step represents the Parulski contribution to the proposed 
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Silber-Parulski combination. Next, the proposed Silber-Parulski 
combination will determine whether the regions of potential occlusion 
represent actual regions of occlusion by examining edges, as the 
prior art and prior rejection have stated, as having a valid focus . If it 

does then a mosaic image is made with the two images by combining 
images." (emphasis added) 

Clearly, the Examiner has explained that the Parulski contribution to the 
proposed combination reference is limited to receiving "two images, and then the 
second image will be subtracted from the first image to provide a single image of a 
'subject,' representing a region of occlusion..." As such, one important issue is whether 
the Silber reference discloses the claimed elements pertaining to determining areas of 
actual occlusion and areas of non-occlusion based on edge discontinuity levels 
determined for each area of potential occlusion. 

In particular, with respect to the issue of "occlusions," the Office Action suggests 
that the Silber reference discloses "determining a level of discontinuity along an 
outer edge of each area of potential occlusion for each image, said level of 
discontinuity indicating an area of actual occlusion where the level of 
discontinuity exceeds a predetermined discontinuity threshold, and said level of 
discontinuity indicating an area of non-occlusion where the level of discontinuity 
is less than the predetermined discontinuity threshold (fig. 5, s730, and as 
explained above)..." (emphasis added). However, Appellant believes that the Office 
Action has misinterpreted the capabilities and features of the Silber reference with 
respect to these alleged occlusion identification capabilities. 

Specifically, as is well known to those skilled in the art, an occlusion in an image 
occurs when an object in an image frame of a particular scene covers or occludes all or 
part of another object in that image frame. In other words, an occlusion in an image 
exists when a region or area of a scene is hidden from view or otherwise covered from 
view by means of some intervening entity or object relative to a viewpoint from which an 
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image of the scene is captured. More simply, the well understood concept of an 
occlusion in an image can be characterized as a region in an image that is obscured 
from view. The specification of the present application specifically defines this idea and 
is wholly consistent with the conventional interpretation of the term "occlusion" as 
recited above. 

For example, as described in paragraph [0056] of the present application (US 
Patent Application No. 2004-0264806 A1 ), "one might have several images of a famous 
monument, but be unable to capture an image of the scene which does not have one or 
more areas of occlusion due to people wandering in and out of the image frame." This 
example is further illustrated by FIG. 4A and FIG. 4B, which illustrate a pictorial 
representation of a sequence of two image frames having areas of occlusions. 
Specifically, as described in paragraph [0107] of the present application, as "illustrated 
by FIG. 4A, image frame 400 illustrates an arch 420 that is partially occluded by two 
people 430. Similarly, as illustrated by FIG. 4B, image frame 410 illustrates the arch 
420, with a background of the image frame being partially occluded by a third person 
440." 

In contrast to the clear definitions and drawings provided in the present 
application, the Examiner has suggested that the interpretation of the term "occlusion" 
offered by the Appellant is only "one interpretation of the word occlusion..." and that if 
"the applicant intends to further define the region of occlusion to mean an area in which 
an object covers another object, the applicant is invited to do so." In response, the 
Appellant did in fact attempt to amend the claim to recite: "wherein an occlusion 
represents an area in any of the images that is at least partially obstructed from 
view by one or more objects. .." However, in the Advisory Action, the Examiner 
responded by refusing to enter the above-quoted amendment and suggesting that the 
"applicant has amended the claims to encompass a different scope and thus would 
require further consideration and search." 
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In view of both the Final Office Action and in view of the Advisory Action, it 
should be clear that the Examiner is interpreting the term "occlusion" in a manner that is 
in direct conflict with the commonly understood meaning of occlusions as specifically 
defined and graphically illustrated in the present specification. Specifically, the 
Examiner has insisted on redefining the term "occlusion" in a manner that is wholly 
inconsistent with both the present specification and drawings, and with the common 
usage of occlusions in the art of image processing as it is generally understood to those 
skilled in the art. 

In fact, in paragraph 7 of the Final Office Action dated May 14, 2007, the 
Examiner states: "The examiner agrees that Silber does not completely provide a 
technique for deciding whether a particular edge represents the boundary of an 
object that is occluding some portion of the scene. .." As such, the meaning of the 
term "occlusion" as specifically defined by the Appellant in the specification of the 
present application and as clearly illustrated by the Appellant in the drawings of the 
present application is of primary concern in this Appeal. 

Specifically, in the Final Office Action, the Examiner argues: 

"... One of the definitions of "occlusion" is "the process of occluding," 
according to dictionary.com, where "occlude" means "to obstruct", or "to 
prevent the passage of." Any region that is masked or not used in further 
processing is an occluded region since the region is obstructed from 
further processing. The region that is not masked is used in further 
processing can also be interpreted as occluded, because it is occluded by 
the mask..." 

The Examiner then uses the circular logic of the above quoted argument in an 
attempt to explain that the Silber reference completely discloses the claimed element of 
"determining a level of discontinuity along an outer edge of each area of potential 
occlusion for each image, said level of discontinuity indicating an area of actual 
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occlusion where the level of discontinuity exceeds a predetermined discontinuity 
threshold, and said level of discontinuity indicating an area of non-occlusion 
where the level of discontinuity is less than the predetermined discontinuity 
threshold..." 

In particular, the Examiner argues that in FIG. 5, step 730, and in paragraph 
[0089], Silber teaches "determining whether each area of potential occlusion in the 
seed image is an actual area of occlusion by examining each area of potential occlusion 
in the seed image to determine whether a level of discontinuity along an outer edge of 
each area of discontinuity exceeds a predetermined threshold." Specifically, in the 
Office Action, dated December 13, 2006, the Examiner stated that this feature of the 
Appellant's claimed system is taught by Silber "since the amount of focus of the line, 
which is equivalent to discontinuous because it is not a solid, clear line, is 
compared to a threshold. . ." (emphasis added). The Examiner then further clarified 
this position in the Final Office Action dated May 14, 2007 by arguing the following: 

"By determining whether a particular edge represents a best level 
focus for that edge, the edge is "highlighted" so to speak as a good area 
to represent the composite image. Therefore, a determination of a 
region of occlusion is accomplished by the evaluation of detected 
edges since the detected edges are evaluated, and a region is 
determined as occluded since certain areas are not further 
processed, the areas that are not of the valid edge, and thus occluded 
from further processing." (emphasis added) 

However, in contrast to the position argued by the Examiner, Appellant 
respectfully suggests that Silber does not disclose the elements suggested by the 
Office Action, regardless of how the term "occlusion" is interpreted . For example, 
as described by Silber in the Abstract, in independent claims 1 , 20, 21 and 22, and 
throughout the specification with respect to Figures 1-13, Silber is directed towards 
compositing images of single objects or "workpieces" in the case where each of the 
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multiple images of the single objects is captured at a different depth of focus. In fact, 
Silber makes use of different depths of focus, in every disclosed and claimed 
embodiment, in order to capture 3D details of the single workpiece such that each 
feature of the single workpiece is captured at various levels of focus. The edges of 
each image are then examined to select those edges that are in best focus, with 
the best focused edges then being used to construct a composite image of the 
single workpiece. Clearly, Silber discloses a process for evaluating edge focus levels 
to determine whether an edge is adequately focused in view of some predetermined 
focus threshold (e.g., the "numerical threshold value" described by Silber). 

For example, in paragraph [0016], Silber specifically summarizes his invention 
as follows: 

"[0016] This invention separately provides systems and methods that 
construct a desirable composite image while suppressing likely 
artifacts arising from out-of -focus features ." 

Further, in paragraph [0089], as cited by the Office Action with respect to Figure 
3, Silber teaches the following: 

"[0089] In step S730, a determination is made whether the selected 
source image analysis result is indicative of a valid edge. For example, if 
the source image analysis result is a numerical edge-focus indicator 
result, this result is compared to a numerical threshold value, where 
only values exceeding the threshold value are taken to be indicative 
of a valid and adeouatelv-focused edge for purposes of constructing 
the composite image. If the source image analysis result is not 
indicative of a valid edge, then operation returns to step S715. Otherwise, 
operation continues to step S735." (emphasis added) 
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Clearly, the focus-based edge detection process disclosed by Silber fails 
completely to provide a technique for "determining... whether regions of potential 
occlusion... represent actual regions of occlusion... or ... regions of non-occlusion, by 
determining a level of discontinuity along an exterior border of each region of potential 
occlusion..." as disclosed and claimed by the Appellant. 

In fact, the invention described by Silber is specifically directed towards 
constructing a composite image from multiple images of a single object or workpiece 
based solely on which edges are in best focus. Consequently, Appellant respectfully 
suggests that any composite image created by the Silber invention from a set of images 
that include one or more actual areas of occlusion would tend to include portions of 
both occluding objects and non-occluded regions of images as a function of edge 
focus levels rather than whether or not an area was actually occluded. In fact, 
Appellant believes that the edges of occluding objects would be favored by the Silber 
system since the edges of any occluding object in a particular image frame would 
necessarily be in better focus than any edges that are actually occluded and would 
therefore not even be visible. 

Clearly, the proposed Silber- Parulski combination reference fails completely to 
provide any technique whatsoever for determining a level of discontinuity along an outer 
edge of each area of potential occlusion for each image, said level of discontinuity 
indicating an area of actual occlusion where the level of discontinuity exceeds a 
predetermined discontinuity threshold, and said level of discontinuity indicating an area 
of non-occlusion where the level of discontinuity is less than the predetermined 
discontinuity threshold. Therefore, in view of the preceding discussion, it is clear that 
the present process, as claimed by independent claim 21 has elements not disclosed in 
the proposed Silber- Parulski combination reference. Consequently, the rejection of 
claim 21 under 35 U.S.C. §1 03(a) is not proper. Therefore, the Appellants respectfully 
traverse the rejection of claim 21 under 35 U.S.C. §1 03(a) in view of the novel language 
of claim 21 . In particular, claim 21 recites the following novel language: 
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"A computer-implemented process for removing occlusions from a 
mosaic image created from a set of images of a scene, comprising: 

inputting a set of two or more images of a scene; 

aligning each of the images to a base image selected from the set 
of images; 

comparing each of the aligned images to identify areas of potential 
occlusion in each of the aligned images; 

determining a level of discontinuity along an outer edge of each 
area of potential occlusion for each image, said level of discontinuity 
indicating an area of actual occlusion where the level of discontinuity 
exceeds a predetermined discontinuity threshold, and said level of 
discontinuity indicating an area of non-occlusion where the level of 
discontinuity is less than the predetermined discontinuity threshold; 

creating an image mask for each image, said image masks 
indicating areas of occlusion and areas of non-occlusion for each image; 
and 

using the image mask for each image for creating a mosaic 
image by replacing areas of actual occlusion in one of the images 
with corresponding areas of non-occlusion from one of the other 
images." (emphasis added) 

VIII. CLAIMS APPENDIX 

The claims listed below provide a complete copy of all claims involved in the 
Appeal: 

Listing of Claims: 

1 (Previously Presented). A physical computer-readable medium having stored 
thereon computer executable instructions for automatically constructing an image 
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mosaic from a set of images of a scene, said computer executable instructions 
comprising: 

inputting a set of images of a scene; 

registering the set of images; 

comparing the set of images to identify areas of difference between the images 
for all images, said areas of difference representing regions of potential occlusion in 
each image; 

determining, for each image, whether regions of potential occlusion in each 
image represent actual regions of occlusion, or whether the regions of potential 
occlusion in each image represent regions of non-occlusion, by determining a level of 
discontinuity along an exterior border of each region of potential occlusion in each 
image; and 

creating a mosaic image by replacing at least one actual region of occlusion in 
one image from the set of images with corresponding regions of non-occlusion from at 
least one other image from the set of images of the scene. 

2 (Original). The computer-readable medium of claim 1 wherein registering the images 
comprises at least one of translating the images, rotating the images, scaling the 
images and warping the images, for aligning the images. 

3 (Original). The computer-readable medium of claim 1 further comprising cropping the 
images after registering the set of images so that all images cover approximately the 
same view of the scene. 

4 (Original). The computer-readable medium of claim 1 further comprising balancing 
the set of images. 

5 (Original). The computer-readable medium of claim 4 wherein balancing the images 
comprises histogram averaging of the images based on corresponding targets for each 
image. 
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6 (Original). The computer-readable medium of claim 4 wherein balancing the images 
comprises white balancing the images based on corresponding targets for each image. 

7 (Original). The computer-readable medium of claim 1 wherein comparing the set of 
images to identify areas of difference between the images for all images further 
comprises comparing corresponding blocks of image pixels between each image to 
determine whether the corresponding blocks match. 

8 (Original). The computer-readable medium of claim 1 wherein comparing the set of 
images to identify areas of difference between the images for all images further 
comprises automatically constructing at least one image mask for identifying non- 
occluded regions and regions of potential occlusion in the set of images. 

9 (Previously Presented). The computer-readable medium of claim 7 determining 
whether the corresponding blocks match further comprises determining whether the 
corresponding blocks match within a predetermined matching threshold. 

10 (Original). The computer-readable medium of claim 1 wherein replacing at least one 
actual region of occlusion in one image from the set of images with corresponding 
regions of non-occlusion further comprises at least one of blending, feathering, and 
pixel averaging, along the edge of the corresponding regions of non-occlusion used to 
replace any actual regions of occlusion. 

1 1 (Original). The computer-readable medium of claim 1 further comprising a user 
interface for excluding one or more actual regions of occlusion from being replaced with 
corresponding regions of non-occlusion. 

12 (Original). A system for removing occlusions from a composite image formed from a 
set of images of a scene, comprising: 

acquiring at least two images of a scene from approximately the same viewpoint; 
aligning each of the images to a base image selected from the set of images; 



Appeal Brief for U.S. Application 10/606,002 
Page 34 of 37 

identifying areas of potential occlusion in each of the aligned images; 
selecting a seed image from the set of images; 

determining whether each area of potential occlusion in the seed image is an 
actual area of occlusion by examining each area of potential occlusion in the seed 
image to determine whether a level of discontinuity along an outer edge of each area of 
discontinuity exceeds a predetermined threshold; 

replacing areas of actual occlusion in the seed image with corresponding non- 
occluded areas from one of the other images in the set to form a composite image from 
the seed image. 

13 (Original). The system of claim 12 wherein the system is embedded in a digital 
camera. 

1 4 (Original). The system of claim 1 2 wherein aligning each of the images to the base 
images comprises applying a geometric transform to each image relative to the base 
image for registering each of the images to the base image. 

15 (Original). The system of claim 14 further comprising individually cropping each 
image so that all images cover approximately the same view of the scene. 

1 6 (Original). The system of claim 1 2 further comprising balancing the images by 
applying any of histogram averaging of the images based on corresponding targets for 
each image, and white balancing of the images based on corresponding targets for 
each image. 

17 (Original). The system of claim 12 wherein identifying areas of potential occlusion in 
each of the aligned images further comprises dividing each image in the set of images 
into a number of pixel blocks, and comparing corresponding pixel blocks between each 
image to determine whether the corresponding blocks match within a predetermined 
threshold. 
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18 (Original). The system of claim 12 wherein identifying areas of potential occlusion in 
each of the aligned images further comprises a pixel-by-pixel comparison of 
corresponding pixels between each image to determine whether the corresponding 
pixels match within a predetermined threshold. 

19 (Original). The system of claim 12 wherein identifying areas of potential occlusion in 
each of the aligned images further comprises automatically constructing an image mask 
for identifying all regions of potential occlusion in the set of images. 

20 (Original). The system of claim 12 wherein replacing areas of actual occlusion in the 
seed image with corresponding non-occluded areas from one of the other images in the 
set further comprises at least one of blending, feathering, and pixel averaging an outer 
edge of the corresponding non-occluded areas with respect to the seed image. 

21 (Original). A computer-implemented process for removing occlusions from a mosaic 
image created from a set of images of a scene, comprising: 

inputting a set of two or more images of a scene; 

aligning each of the images to a base image selected from the set of images; 

comparing each of the aligned images to identify areas of potential occlusion in 
each of the aligned images; 

determining a level of discontinuity along an outer edge of each area of potential 
occlusion for each image, said level of discontinuity indicating an area of actual 
occlusion where the level of discontinuity exceeds a predetermined discontinuity 
threshold, and said level of discontinuity indicating an area of non-occlusion where the 
level of discontinuity is less than the predetermined discontinuity threshold; 

creating an image mask for each image, said image masks indicating areas of 
occlusion and areas of non-occlusion for each image; and 

using the image mask for each image for creating a mosaic image by replacing 
areas of actual occlusion in one of the images with corresponding areas of non- 
occlusion from one of the other images. 
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22 (Original). The computer-implemented process of claim 21 wherein the computer is 
integral to a digital camera, and wherein the composite image is formed automatically 
from images of a scene captured using the digital camera. 

23 (Original). The computer-implemented process of claim 21 wherein replacing areas 
of actual occlusion in one of the images with corresponding areas of non-occlusion from 
one of the other images further comprises blending pixels along an outer edge of the 
corresponding areas of non-occlusion with pixels surrounding the areas of actual 
occlusion being replaced. 

24 (Original). The computer-implemented process of claim 21 wherein replacing areas 
of actual occlusion in one of the images with corresponding areas of non-occlusion from 
one of the other images further comprises feathering pixels along an outer edge of the 
corresponding areas of non-occlusion with pixels surrounding the areas of actual 
occlusion being replaced. 

25 (Original). The computer-implemented process of claim 21 wherein replacing areas 
of actual occlusion in one of the images with corresponding areas of non-occlusion from 
one of the other images further comprises averaging pixels along an outer edge of the 
corresponding areas of non-occlusion with pixels surrounding the areas of actual 
occlusion being replaced. 

26 (Original). The computer-implemented process of claim 21 further comprising 
balancing the set of images after registering the set of images. 

27 (Original). The computer-implemented process of claim 26 wherein balancing the 
images comprises histogram averaging of the images based on corresponding targets 
for each image. 
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28 (Original). The computer-implemented process of claim 26 wherein balancing the 
images comprises white balancing the images based on corresponding targets for each 
image. 

IX. EVIDENCE APPENDIX 

None 

X. RELATED PROCEEDINGS APPENDIX 



None 
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